Unsupervised Learning from URL Corpora

This paper illustrates the utility of URL information in unsupervised learning. We outline the motivation behind the usage of URL information upfront, and present two techniques for unsupervised learning from URL corpora. First, we devise a similarity measure for URL pairs putting down the intuitions behind the same and verify its goodness by using it for clustering. Further, we outline a method for keyword identification using the similarity measure. Then, we explore the usage of character N-grams for flat clustering of URL corpora. The motivation to keep URLs compact forces the usage of a lot of variations of the same word, which is a very unique kind of forcibly inserted noise. N-gram based models are very tolerant to such noise and are very compact too. We then compare the two similarity measures using a rank correlation measure, which reveals that their similarity is very subjective on the corpus used and depends on the distinctiveness of the clusters in the corpus. Given that URLs are small entities, our techniques are magnitudes faster than unsupervised techniques on full-text corpora and require far less information than the latter. To the best of our knowledge, this is the first attempt on unsupervised learning from URL information.